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The work of this Panel for the Evaluation of 
Experimental Literacy Projects was the assessing of how the pattern 
of evaluation it had previously formulated had been applied in the 
Experimental World Literacy Programme (EWLP) , In the judgment of the 
Panel, by 1974 evaluation specialists will be able to present a 
report that will enable Member states to make a reasonably 
well-founded judgment on the value of functional literacy and on the 
efficiency of various means of achieving it. it is stressed that the 
full effects of a successful functional literacy program may not been 
seen for a decade or more. It is also the opinion of the Panel that 
by 1972, it should be possible to see on a limited scale the extent 
to which some programs have changed traditional practices in 
agriculture and industry. In addition to use of proposed indicators 
in the Standardized Data Reporting System to evaluate the EWLP, it is 
recommended that other material be collected also on qualitative and 
organizational aspects of the program. The Panel also urged that all 
projects work in close cooperation with the responsible authorities 
to facilitate the eventual transfer of pilot projects into national 
structures. The Panel felt that functional literacy works best when 
closely related to specific development objectives. The distribution 
of a costing manual to all projects and more research by universities 
are seen as needs. (DB) 



TITLE 

INSTITUTION 

REPORT NC 
PUB DATE 
NOTE 

EDRS PRICE 
DESCRIPTORS 

IDENTIFIERS 

ABSTRACT 



„v,’ i ' i . 




r. 




CM 



ITS 



rH 







Limited distribution 



U S DEPARTMENT OF HEALTH. 
EDUCATION S. WELFARE 
OFFICE OF EDUCATION 
THIS DOCUMINT HAS BEEN REPRO 
DUCID EXACTLY AS RECEIVED FROM 
THE PERSON OR ORGANIZATION ORIG- 
INATING IT POINTS OF VIEW OR ODN 
'IONS STATED DO NOT NECESSARILY 
REPRESENT OFFICIAL OFFICE OF EDU 
CATION POSITION OR POLICY 



ED.71/C0NF.1, 3/5 
PARIS, Lth October 1971 
Original: English 



UNITED NATIONS EDUCATIONAL, SCIENTIFIC 
AND CULTURAL ORGANIZATION 




UJ 



Fourth Meeting of the Panel for Evaluation 
of Experimental Literacy Projects 



(Unesco House, Paris, SOth-ZLth September 1971) 



FIN AL REPORT 



1, The Fourth Meeting of the Panel for the revaluation of Experimental 
Literacy Projects was inaugurated at Unesco Headquarters on 20th September 
1971. The Panel, established in 1967 for a period of five years to advise the 
Director-General on all problems of evaluation in the experimental literacy 
projects, held its first and second meetings in Unesco House, Paris, from 
llth-l5th December 1967 and 2nd«6th December 1968. Panel members also v * * . s 
participated in a special meeting in Unesco House from lst-5th December 1969 ,/V 
to consider the problems in the Experimental World Literacy Programme and to , v . 
make recommendations for overcoming them. The third meeting was held in Iran 
from 8th-I7th September 1970. 



2. At the beginning of the Meeting Panel members and other participants were 
informed that Mr. Oscar Vera, who had played a distinguished role in the Panel 
since its inception, had died in May 1971. Mr. J, C. Cairns, representing the 
Director-General, and Mr. C. 3. Beeby, speaking for the Panel, paid tribute 
to Mr. Vera’s outstanding qualities and his important contribution to Unesco* s 
work. It was unanimously decided that a letter of sympathy would be sent to 
Mrs. Vera by the Panel to convey their condolences. 
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3, The Meeting was attended by six members of the Panel. Mr. S. M. Sharif 
was unable to attend for personal reasons, Mr. Rene Dumont had resigned 
during the previous year j Panel members, while appreciating the reasons for 
his decision, stressed the value of the contribution he had made. The Panel 
also recorded its deep appreciation of the outstanding dedication and leader- 
ship given to the B.W.L.P. by Mr# A, Deleon, who, since December 1970, has been 
seconded as Executive Secretary of the International- 0^ 

Development of Education. Two representatives of U.N.D.P., representatives of 
three diversities, the national director from Sudan, national experts fr^^} 
Algeria, Ecuador, Ethiopia, Mali and Tanzania, Chief Technical Advisers from 
Afghanistan, Algeria, Mali and Tanzania, the Co-Director of A.S.F.E.C. and 
evaluators from C.R.E.F.A.L., Ecuador^ Ethiopia,; Guinea, Iran, Arab Republic 
of Egypt , Sudan, Tanzania and Venezuela, as well as an international expert 
from Venezuela, attended the Meeting .(for list of participants, see Appendix 1) . 

U. The Panel decided unanimously that Mr. C. E. Beeby, Mr. A, Bouhdiba and 
Mr. D. Acquah act asFChairman, Vice-Chairman and Rapporteur, respectively, for 
the Fourth Meeting (for the Agenda, see Appendix 2). 
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5. Ml*. J, C. Cairns, Acting Director, Department of Out-of -School Education, 
presented a general report on the substantial progress made, both qualitively 
and quantitatively, in the E.W.L.P. since September 1970. For training of 
senior national counterparts the new concept of field operational seminars had 
proved extremely valuable and was being extended! arrangements had also been 
developed with three universities to provide specialised training facilities. 

A significant turning point in the overall evaluation strategy had been reached 
at the 1970 Evaluation Panel and the associated experts 1 meeting j since then 
three experts 1 meetings have been held at Headquarters in May, July and 
September 1971, leading to systematic, professional developmant in evaluation 
designed to prepare for a preliminary progress report in 1972 and a more 
substantial report in 197U. : 

6. Among the important priorities facing the E.W.L.P. , Mr. Cairns mentioned? 
(a) the need to develop effective relationships between the experimental 
projects and existing national jhsogrammes, to determine the ministries or 
organizations which would take over functional literacy responsibilities on 
completion of the projects, and to make effective arrangements for such take- 
overs during the final stages of the experimental projects! 0>J the solving 

of problems of teacher-training, support, organization and administration as 
projects moved from the small-scale laboratory stage to the expansion stage! 

(o) the need to devote increasing attention during the final stages of projects 
to post-literacy planning! (d) the obtaining of additional sources for large- 
scale funding. 

7. Mr. Platt, Director of the Department of the Planning and Financing of 
Education, pointed out the relevance of functional literacy as a potential 
answer to the urgent need of developing new educational strategies. He con- 
sidered that, for evaluation purposes, monetary concepts alone would not 
provide the required answers, but changes in working behaviour and in life- 
style, together with possible side-effects not quantifiable, should form a 
basis for comprehensive _ evaluati©n of functional literacy programmes. 

8. The work of this Panel differed in some ways from previous meetings in 
that it assessed how the pattern of evaluation it had laid down earlier had 
been applied in the E.W.L.P. The last two meetings had been devoted more to 
discussions of basic principles and the formulation of general patterns. 

9. Evaluation ? The Panel reviewed the application in the pro jects of its 

recommendations concerning evaluation. It noted with satis- 
faction that two important meetings of evaluation experts (3rd-8th May 1971 
and I5th-17th September 1971) led to the establishment of a Standardized Data 
Reporting System as well as a precise reporting: schedule for 1972 • The Panel 
also noted the important achievements in developing a common approach to the 
construction of various ijistruments while still maintaining sufficient freedom 
in each project to meet specific local requirement s. On the ba si s of two 
docum^ts|^*eflented to it (ED.7^GOT^ii3/3 and lbA^2^ 

hearing reports by all project evaluators, the Panel is distinctly impressed 
with the gheat ii^eyemeiit in evaluation since 1969 . The Panel recognizes a 
new professional qualij^y in much of the i work currently undertaken in evaluation, 
w^ be the most encouraging davelopment in this field ewer 

the past year. The meetings ; of the .experts in evaluation 1 and methodology 
held during the year have clearly contributed to this and to the growth of 
a group spirit which was obvious in the discussions. This Meeting has made 
it clear that, with the increased experience of the evaluators, a certain 
quantity of data in a systematic form w ill be availabla from the projects by 
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early in 1972, but that anything like a comprehensive report on the results of 
the programmes as a whole cannot be expected until 197U, 

10. Despite the good progress made in the evaluation of many projects, the 
Panel feels that some of the evaluation experts may still underestimate the 
amount of work involved in meeting the commitments they made in September and 
reaffirmed at this Meeting, They have limited resources of time, funds and 
personnel, and the kinds of data they contemplate collecting can pile up at 

an alarming rate and can put intolerable strain on staff, both in the field ard 
at- Headquarters, The Panel suggests that, if the experts find themselves in 
danger of being overwhelmed by data, they establish a system of priorities 
that will enable them to process a limited quantity of data properly and 
present them in a useable form early in 1972, 

11. Although no definite date has yet been set for the presentation of the 
first report on the projects as a whole, it seems likely that Member States will 
expect a progress report for the consideration of the General Conference in 
1972, particularly on the projects that will, by then, have been operating for 
five years. The information that will be available at that time cannot 
possibly answer all the questions that have been raised concerning functional 
literacy but it should be possible by then uo give some significant indication 
of the degree of success that may be expected of the projects, 

12. The Panel is confident that, if the evaluation specialists can maintain 
the rate of professional growth they have shown over the past twelve months, 
they will, by 197^4, be able to present a report that will enable Member States 
to make a reasonably well-founded judgment on the value of functional literacy 
and on the efficiency of various means of achieving it. It must be stressed, 
however, that the full effects of a successful functional literacy programme 
may not be seen for a decade or more. In a rapidly changing world the final 
test of a functional literacy programme is not just whether it enables people 
to do their present job better but whether it helps them to meet constantly 
changing demands, both vocational and social, 

13« Mien the first projects were started in 196? there was a general 
expectation that, within four or five years, they would show a measurable 
effect on productivity. It is now quite clear that, except in a few specific 
cases and on a very narrow front, this is too much to expect, particularly as 
most projects took far longer to become operational than had been foreseen. 
Nevertheless, by 1972 , it should be possible to see on a limited scale the 
extent to which some prograirmes have changed traditional practices in agri- 
culture and industry, Experts in these fields should be able to make fairly 
accurate forecasts of the effects these changes in practice will have on 
production, without waiting for actual production figures that will be dependent 
in any year on the vagaries of c|Jjiate or on changes of management in industry. 
So the Psmeif strongly recommends that Indicator 2,13 (adoption of recommended 
practices) be no longer optional but be considered as an essential indicator in 
programmes where it is relevant. Some participants in the discussion main- 
tained, moreover this Iridieatoyiis a far more reliable measure of 

changed attitudes than is any instrument for measuring attitude change# 

1U. If at all possible, changes in public knowledge of functional programmes 
and changes in the attitudes of employers:, civic headers, government ^ officials 
and others concerned with carrying out functional literacy programmes should 
also be subject to some type of assessment. 
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15. The Panel then went on to consider whether the proposed indicators in 
the Standardized Data Reporting System, even when they are fully covered, will 
be adequate in themselves to provide answers to all the questions on which 
Members States will expect guidance from the evaluation programme. It is quite 
clear that more will be expected of evaluation than these indicators alone 
can offer and that valuable experience will be lost unless a sy sterna tic effort 
is made to gather other information on qualitative and organizational aspects 
of the programme. The Panel, therefore, recommends that project staff be asked 
to collect other material, not covered by the standard indicators, which will: 

a) provide a continuing process evaluation of qualitative and 
organizational aspects of each project; 

b) enable them to present a report on these aspects as a part of 
the final evaluation of the project; 

e) provide information which will allow informed judgments to 
be made on the most effective inputs of the projects (e.g. 
the most receptive participants, the best organizational 
framework, etc,); 



d) assist an outside team to make a clinical case study of each 
project at an appropriate moment before the experimental 
phase ceases, 

16. To this end it is suggested that the Secretariat at Headquarters take 
steps to prepare a paper to guide the project staff in the collection of 
such material. It is suggested also that universities may have a useful part 
to play in this type of evaluation. Obviously, the cost of sending an outside 
team of, say, two people to make such a case study in each survey, will be 
considerable, and it is suggested that this might be the type of activity in 
which the U N 0 D.P. might be especially interested, 

17. Mr. Luke, representative of U.N.D.P. at the Meeting, stated that his 

Organization would welcome the new emphasis now being placed on evaluation of 
results as opposed to evaluation for programme support. Discussions on the 
Standardized Data Reporting System reflected a blend of caution and determi- 
nation on the part of various evaluators concerned, which augured well for the 
evaluation exerci.se now proposed. Hopefully, the results obtained by March 
1972 would be sufficiently representative to influence significantly the 
future of the projects, .c,,;'. 

18. Although, of course, Mr. Luke could not commit his Headquarters, he 

believed U.N.D.P. would be dispo sed to favour increased support for the 
evaluatiori aspects of the functional literacy project s presently assisted by 
U.H.D.P,, if this were found to be necessary. ... 

19 . Transition from Pilot Projects to Matlonal Programsiie s i The Panel had an 

•; . .. ; .. . . \ . opportunity to 

consider once more the relationship between pilot pro j ects and national 
programme s . This question takes ; on a dimension pf special urgency because 
several pilot projects will come to a Close at the end of 1972. ‘ . 
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20, After reviewing, with the experts present at this Meeting, the steps 
taken to integrate pilot pro jects’with national structures, the Panel concluded 
that they were further advanced than the Panel expected and give promise that, 
in many cases, the transfer of projects is being well prepared and will 
proceed smoothly, 

21, The Panel recommends that steps be taken to ensure the presence of at 
least one international expert beyond the pilot stage of the project. This 
will permit the carrying out of post-literacy programmes and any additional 
evaluation still considered essential. International experts could be 
requested under the new country programming scheme. The Panel urges that all 
projects which have not done so take immediate steps to work in close co- 
operation with the responsible authorities to facilitate the eventual transfer 
of pilot projects, 

22. Definition of functional literacy i The Panel took note of the definition ? 

of functional literacy as stated in 

the Draft Programme and Budget for 1971-72 (paragraphs 523-327). The Panel 
underlined once more the need for flexibility in any definition of the 
concept of functional literacy. This is necessary to permit the inclusion of 
many different national conditions in a continuing clarification of the idea 
of functional literacy. The Panel confirms its statement on functional 
literacy appearing in its report of the Third Meeting (paragraph 21*')... Some 
members wished to underline the important contribution of functional literacy 
to the improvement of the quality of life and felt the concept should not be 
confined to economic consideration, 

23. Functional literacy should serve not only in the achievement of specific 
and isolated development objectives but should contribute to creating an 
environment in which literate adults can work most effectively towards 
national development goals. However, experience has shorn that functional 
literacy seems to work best when closely related to specific development 
objectives. 

21*, Costing of functional literacy ! The Panel recognised the many difficulties 

of a methodological nature, but considered 
the costing of functional literacy to be of great importance* The Panel, 
therefore, urged that initial steps be taken by the Secretariat to provide for 
the distribution, of a costing manual to all projects. This guide would permit 
the colic ction of information on the cost of functional programmes and thus 
form an essential element 'in the final evaluation of the economic viability 
of functional literacy. The Panel suggests that the Secretariat work in 
collaboration with I.I.F.P. during the coming year on this important question. 

25, Research and relations with universities ; The Panel was glad to note 

:• . £ v ,,, ' ’ evidence of some co-operation 

with universities in the past year but felt that more could still be done. 

It expressed the hope that steps be taken at a. national level to involve 
universities and research institutions in carrying out research of value to 
the E.V.L.P. The Panel is aware that many academic institutions are either 
disinclined or unable to provide the necessary research support. Difficulties 
may have arisen sometimes because universities have emphasised fundamental 
research, whereas the first requirement of projects is research that will 
give them operational support. . . . . 
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26. The Panel recommends that national universities be encouraged to 
complement, whenever possible, the work of project teams with particular stress 
oni (a) research in methodology! (b) training in research,; (c) analysis and 
treatment of evaluation data; (d) all aspects of research requiring a special 
understanding of the culture in which projects operate. It further targes 

that contacts be established at local levels, either on a person-to-person 
basis or through formal institutional channels and points out that the greatest 
help will often come from specialised institutions, such as agricultural 
colleges and technical institutions. It was suggested by one participant- 
that, in some cases, the most effective co-operation can be achieved by 
linking project research with some on-going activities in the universities. 
Finally, the Panel recommends that the Secretariat prepare a statement on a 
clear strategy on research, taking into consideration the long-term require- 
ments of the R.W.L.P., including research to he carried out by national 
institutions after the closing of pilot projects, 

27. One of the essential difficulties of research in the past has been the 
lack of adequate financial resources. Therefore, the Panel was particularly 
interested in Hie statement of the U.N.D.P. representative that financing of 
research may be possible within the individual projects 1 budgets if their 
funds permit, or alternatively, a separate research project might be considered, 
to be charged against the countries* Indicative Planning Figure if the governs. ; 
ments so wish. Funds for "global" projects, on the other hand, are, at 
present, limited to 1 % of the total U.N.D.P. resources available. Support 
from governments for a '’global" or even an ” inter-regional" research project 
would need to be carefully weighed before it were submitted to the Administration 
for consideration. If the Programme Working Group were favourably disposed 
towards a project of this land, the endorsement of the proposal by three or 

four governments would still need to be obtained before the project requests 
could be processed further. 

28. Follow-up on recommendations of the last Panel Meeting ; After hearing the 

reports of the 

representative of the Directa’ -General and in light of its discussions 
throughout this Meeting, the Panel is of the opinion that some of its 
recommendations formulated at the Third Session had been adopted in full and 
most reasonable progress had been made towards carrying out most of the rest, 

Mr. Cairns had stressed some of the early difficulties of many of the pilot 
projects due to unrealistic targets in the plans of operation. It was often 
very difficult to readjust those targets and to revise the plans of operation. 

Ref erring to these difficulties experienced by the S.W.L.P., Mr. Luke stated 
that the introduction of the word "pilot" into the title of some of the projects 
had seemed, to the U.N.D.P,, to reflect a moving away from a limited experi- 
mentation operation, which in its view has been the objective.; of these projects, 
towards expansion. Changes iri the directieaa of the pi^jects ha^, . therefore, 
been questioned, formally, modifications of plans of operation through 
Adjustment Advice and Amendments were fairly routine. He thought that, at 
this stage, there would be little likelihood of hegotiatiohs for changes in 
the plans of operation being protractedy if they were proposed in the spirit 
of experimentation and in the interests of furthering project evaluation, v 

29 • Finally, the Panel supports the Secretariat* s plan to convene a meeting 
of methodology experts in the coming year. ; V --Sjw)i arinie be useful, if 

properly prepared, in stressing once more the experimorital aspects of the 
e.tj.l.p. • • ; - 
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Appendix 1 
LIST OF PARTICIPANTS 



Members of the : Permanent Panel 



Mr. D. ACQUAH 
Mr. C. E, BEEEY 
Mr. A. BOUKDIBA 
to, C. LOOMIS 
to, J, LtJTYNSKI 
to. Shib IC. ULTRA 



Evaluation Specialists 



Expert 



Afghanistan 

Algeria 

A.S.F.E.C. 

C.R.E.F • A , L . 

Ecuador 

Ethiopia 

Guinea 

Iran 

Mali 

A.R, of Egypt 
Sudan 

Tanzania 

Venezuela 



to. A, Ikene 



Mrs. Mariana Almeida 
Mr. La,eke Abbay 



to. Isa iCone 

to, Abdel Aziz Ahmed 
(National Director) ; 
to, Emmanuel Mbalcile 



to, H. Mobarak, C.T.A, 

Mr, J, Mourad, C.T.A, 

Mr. M, Hamza, Co-Director 
to. A, Tobelem, Evaluator 
Mr. P, Uollrabe, " 

to, B, N. Singh, 11 

Mr. R, Couvert, 11 

( to. M, Bazany, " 

( to. H.-D, iCaufmann 
to, B. Dumont, C.T.A, 
to. J, H, Saksena, Evaluator 
Mr. A. Sammak, Evaluator 

( to. Vishnyakov, C.T.A. 

( to. E, Mezzaoapa, Evaluator 
to. K. Neijs, Evaluator 



Representatives of U.M.D.P . i 

Indiana University (U.S.A ,. ) t 
Nice University (France ) i 
L.I.R.S ,H. A. , Spoleto (Italy) i 



Mr, Kenneth Luke 
to, Fattabhi Raman 

to, M. Ohiappetta 

Mr. M. Oriol 

Mr. R. Cali si 




Monday, 20th September 
morning 



afternoon 



Tuesday, 21st September 
morning and afternoon 

Wednesday, 22nd September 
morning 

afternoon 

Thursday, 23rd September 
morning 

afternoon 

Friday, 2ljth September 
morning 



0 
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Appendix 2 

AGENDA 



1* Statement and Report on the progress' of 
the E.W.L.P. since the last session by the 
representative of the Director-General 

2, Statement by the Director of the Department 
of Educational Planning on the importance of 
evaluation of functional literacy 



3. Presentation by the Secretariat on the 
results of three experts* meetings.* 

a) 3rd-8th May 1971 (Doc, ED-7l/C0NF.i*3/3) 

b) |th-7th July 1971 (Doc, EDA ,1797/27 .07 
and Doc, EM . l85b/2 , 8 ) 

c) 15 th -17th September 1971 (Doc, EB\/ 

2265(a)/20.09 and Doc, ED A/2 265(b)/ 
20,09) - 



U« Review of evaluation work and consideration 
of the proposed working schedule 



5- General appraisal of the S.D.R.S, 

6, Transfer of pilot projects to national 
programmes 



7* Costing of functional progranmes 



8. Research and relationship with universities 



9* Review of follow-up action of its recommend* 
ations of last year ."-i- -. 1 .?;'. ; 



ID. Approval of the Final Report 
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